System and method of controlling external apparatus connected with device

ABSTRACT

A method of controlling an external apparatus includes receiving a user input information; obtaining apparatus information regarding a plurality of external apparatuses; selecting one or more external apparatuses, from the plurality of external apparatuses, which is communicable with and controllable based on the user input information; generating a control information for controlling the one or more external apparatuses based on a user&#39;s input and the apparatus information; and transmitting a control command to the one or more external apparatuses, the control command being generated based on the received control information.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This is a Continuation of U.S. application Ser. No. 14/022,556, filedSep. 10, 2013, which claims priority from the Korean Patent ApplicationNo. 10-2012-0099739, filed Sep. 10, 2012, in the Korean IntellectualProperty Office. The disclosures of the above-named applications areincorporated herein in their entireties by reference.

BACKGROUND 1. Field

Apparatuses and methods consistent with exemplary embodiments relate tocontrolling an external apparatus connected with a device according to auser's intention.

2. Description of the Related Art

Due to development of natural language analysis technologies and mobiletechnologies, a device may analyze information input by a user and mayoperate according to the user's intention. However, even though a devicemay be able to operate according to a user's intention, it is difficultfor the device to utilize various connectable or device-compliantapparatuses. As such, it is difficult to control a device by effectivelyreflecting various true intentions of a user.

Accordingly, there is a need for methods and apparatuses for obtainingvarious types of condition information by using a device and an externalapparatus connectable with the device and for operating the device anddifferent external apparatuses by effectively reflecting variousintentions of a user.

SUMMARY

Exemplary embodiments may address at least the above problems and/ordisadvantages and other disadvantages not described above. The exemplaryembodiments are not required to overcome the disadvantages describedabove, and an exemplary embodiment may not overcome any of the problemsdescribed above.

One or more of exemplary embodiments provide a system and method ofcontrolling an external apparatus connected with a device by determininga user's intention based on user input information, and controlling thedevice and the external apparatus according to the user's intention.

According to an aspect of an exemplary embodiment, there is provided amethod of controlling an external apparatus performed by a device, themethod including providing to a server user input information used todetermine a user's intention; checking at least one external apparatuscontrollable by the device; providing apparatus information of thechecked external apparatus to the server; receiving from the servercontrol information about the external apparatus, wherein the controlinformation is generated based on the user input information and theapparatus information by the server; and transmitting a control commandto the at least one external apparatus based on the received controlinformation, wherein the control information is generated by the serverbased on the user's intention.

A condition for controlling the external apparatus may be determined bythe server based on the user's intention, and the control informationmay be generated by the server based on the determined condition.

The control information may include information for checking whether thedetermined condition is satisfied, and information for controllingoperation of at least one of the device and the external apparatus ifthe determined condition is satisfied.

The method may further include obtaining condition information relatedto the determined condition, and the transmitting of the control commandmay include transmitting the control command to the external apparatusif it is checked that the determined condition is satisfied based on theobtained condition information.

The obtaining of the condition information may include obtaining thecondition information from the at least one external apparatus.

The condition information may include information about at least one ofa location of the device, an environment around the device, weather,time, temperature, humidity, and illumination at the location of thedevice, and traffic at a preset location.

The user input information may include information about at least one oftext input by the user, a voice of the user, and a gesture of the user.

The apparatus information may include information about at least one ofan identification value of the external apparatus, a type of theexternal apparatus, and a command used to control the externalapparatus.

An external apparatus related to the user's intention may be selectedfrom among a plurality of external apparatuses, and the control commandmay be a command for operating the selected external apparatus.

The method may further include, if an external apparatus is additionallyconnected with the device, providing apparatus information of theadditionally connected external apparatus to the server, and the controlinformation including a control command about the additionally connectedexternal apparatus may be generated by the server.

According to another aspect of an exemplary embodiment, there isprovided a method of providing control information for controlling anexternal apparatus to a device performed by a server, the methodincluding receiving user input information from the device; receivingfrom the device apparatus information of at least one external apparatuscontrollable by the device; determining a user's intention based on theuser input information; generating control information about theexternal apparatus based on the determined user's intention and theapparatus information; and providing the control information to thedevice.

The method may further include determining a condition for controllingthe external apparatus, based on the user's intention, and thegenerating of the control information may include generating the controlinformation based on the determined condition.

The control information may include information for checking whether thedetermined condition is satisfied, and information for controllingoperation of at least one of the device and the external apparatus ifthe determined condition is satisfied.

Condition information related to the determined condition may beobtained by the device based on the control information, and the controlcommand may be transmitted from the device to the external apparatus ifit is checked that the determined condition is satisfied based on theobtained condition information.

The condition information may be obtained by the device by using the atleast one external apparatus.

The condition information may include information about at least one ofa location of the device, an environment around the device, weather,time, temperature, humidity, and illumination at the location of thedevice, and traffic at a preset location.

The user input information may include information about at least one oftext input by the user, a voice of the user, and a gesture of the user.

The apparatus information may include information about at least one ofan identification value of the external apparatus, a type of theexternal apparatus, and a command used to control the externalapparatus.

An external apparatus related to the user's intention may be selectedfrom among a plurality of external apparatuses, and the controlinformation may be information for operating the selected externalapparatus.

The receiving of the apparatus information may include, if an externalapparatus is additionally connected with the device, receiving apparatusinformation of the additionally connected external apparatus from thedevice, and the generating of the control information may includegenerating the control information including a control command about theadditionally connected external apparatus.

According to another aspect of an exemplary embodiment, there isprovided a device including a memory for storing at least one program;and a processor for controlling an external apparatus by executing theat least one program, wherein the at least one program includes commandsinstructing to check at least one external apparatus controllable by thedevice; provide apparatus information of the checked external apparatusto a server; receive from the server control information about theexternal apparatus, which is generated based on the user inputinformation and the apparatus information; and transmit a controlcommand to the at least one external apparatus based on the receivedcontrol information, and wherein the user input information may be usedwhen the server determines a user's intention, and the controlinformation may be generated by the server based on the user'sintention.

A condition for controlling the external apparatus may be determined bythe server based on the user's intention, and the control informationmay be generated by the server based on the determined condition.

The control information may include information for checking whether thedetermined condition is satisfied, and information for controllingoperation of at least one of the device and the external apparatus ifthe determined condition is satisfied.

According to another aspect of an exemplary embodiment, there isprovided a computer-readable recording medium having recorded thereon acomputer program for executing the above method.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects will become more apparent by describingcertain exemplary embodiments, with reference to the accompanyingdrawings, in which:

FIG. 1 is an overall schematic diagram of a system for controlling anexternal apparatus via a device, according to an exemplary embodiment;

FIG. 2 is a flowchart of a method of controlling an external apparatusby a device, according to an exemplary embodiment;

FIG. 3 is a flowchart of a method of controlling an external apparatusby a device, according to an exemplary embodiment;

FIG. 4 is a flowchart of a method of controlling an external apparatusby a device, according to an exemplary embodiment;

FIG. 5 is a flowchart of a method of controlling an external apparatusby a device, according to an exemplary embodiment;

FIG. 6 is an operation table, according to an exemplary embodiment;

FIG. 7 is a schematic diagram of controlling an external apparatus,according to an exemplary embodiment;

FIG. 8 is a schematic diagram of controlling an external apparatus,according to an exemplary embodiment;

FIG. 9 is a schematic diagram of controlling an external apparatus,according to an exemplary embodiment;

FIG. 10 is a block diagram of a device according to an exemplaryembodiment;

FIG. 11 is a block diagram of a condition information obtainer accordingto an exemplary embodiment;

FIG. 12 is a block diagram of a device according to an exemplaryembodiment; and

FIG. 13 is a block diagram of a server according to an exemplaryembodiment.

DETAILED DESCRIPTION

Certain exemplary embodiments are described in greater detail below withreference to the accompanying drawings.

In the following description, the same drawing reference numerals areused for the same elements even in different drawings. The mattersdefined in the description, such as detailed construction and elements,are provided to assist in a comprehensive understanding of exemplaryembodiments. Thus, it is apparent that exemplary embodiments can becarried out without those specifically defined matters. Also, well-knownfunctions or constructions are not described in detail since they wouldobscure exemplary embodiments with unnecessary detail.

Throughout the specification, it should be understood that when anelement is referred to as being “connected with” another element, it canbe directly connected with the other element, or electrically connectedwith the other element while intervening elements may also be present.Also, when a part “includes” or “comprises” an element, unless there isa particular description contrary thereto, the part can further includeother elements, not excluding the other elements.

Expressions such as “at least one of,” when preceding a list ofelements, modify the entire list of elements and do not modify theindividual elements of the list.

FIG. 1 is an overall schematic diagram of a system for controlling anexternal apparatus 2000 via a device 1000, according to an exemplaryembodiment.

As illustrated in FIG. 1, the system includes the device 1000, one ormore external apparatuses 2000 connected with the device 1000, a server3000, and a network 4000.

Referring to FIG. 1, the device 1000 may generate user input informationbased on a user input, and may provide the user input information,apparatus information of the external apparatus 2000, and conditioninformation to the server 3000 such that the external apparatus 2000 maybe controlled according to a user's intention.

The device 1000 may receive a text input or a voice input of the user,may generate the user input information based on the received input, andmay provide the generated user input information to the server 3000. Thedevice 1000 may check the external apparatus 2000 connectable with thedevice 1000, and may obtain and provide the apparatus information of theexternal apparatus 2000 to the server 3000. Furthermore, the device 1000may obtain the condition information via the device 1000 or the externalapparatus 2000, and may provide the obtained condition information tothe server 3000. The device 1000 may obtain the condition informationbased on control information to be described below, but is not limitedthereto.

The server 3000 may analyze the user's intention based on the user inputinformation, and may generate and provide the control information to thedevice 1000 such that the device 1000 may operate according to theuser's intention and that the external apparatus 2000 may be controlledby the device 1000. The server 3000 may determine the user's intentionby analyzing the user input information, and may determine a conditionfor controlling the external apparatus 2000 in such a way that theuser's intention is satisfied. The server 3000 may check whether thedetermined condition is satisfied and may generate the controlinformation for controlling operations of the device 1000 and theexternal apparatus 2000.

The device 1000 may receive the control information from the server3000, and may transmit a control command to some external apparatuses2000 based on the control information.

The device 1000 is an apparatus capable of transmitting and receivinginformation to and from the server 3000 and of transmitting the controlcommand to the external apparatus 2000, and may be, for example, asmartphone, a mobile phone, a personal digital assistant (PDA), a laptopcomputer, a media player, a global positioning system (GPS) apparatus,or another mobile or non-mobile computing apparatus, but is not limitedthereto.

The external apparatus 2000 may be an apparatus connectable with andcontrollable by the device 1000, and/or capable of at least partiallysharing functions provided by the device 1000. The external apparatus2000 may be a stand-alone apparatus capable of independently executingan application, but is not limited thereto. The external apparatus 2000may include an input apparatus, an output apparatus, or a controlapparatus, for example, a home appliance such as a smart TV or an airconditioner, a security camera, a recorder, a mobile phone, a personalcomputer (PC), a pillow vibrator, a motor vehicle, a navigator, amicrophone, a speaker, a pedal, a joystick, a musical instrument (e.g.,a plano, an organ, an electronic keyboard, a guitar, a violin, or acello), a game manipulator, a doll, a medical appliance, sportingequipment, a camera, or a sensor.

The server 3000 may be a service providing server for providing thecontrol information for allowing the device 1000 to control the externalapparatus 2000 according to the user's intention, but is not limitedthereto.

The server 3000 may be a cloud server, for example, a PC of the user. Ifthe server 3000 is a cloud server, the server 3000 may include apredetermined intelligence engine, may analyze the user's intention viathe intelligence engine, and may transmit the control information forcontrolling the external apparatus 2000 to the device 1000. Otherwise,if the server 3000 is a client server, the server 3000 may receivepredetermined information from a separate service providing server(e.g., a weather information providing server or a traffic informationproviding server).

The network 4000 may be implemented as a wired network such as a localarea network (LAN), a wide area network (WAN), or a value added network(VAN), or a wireless network such as a mobile radio communicationnetwork, a near field communication (NFC) network, or a satellitecommunication network. The network 4000 may be a comprehensive datacommunication network for allowing appropriate communications betweenthe network components illustrated in FIG. 1, and may include wiredInternet, wireless Internet, and/or a mobile wireless communicationnetwork.

FIG. 2 is a flowchart of a method of controlling the external apparatus2000 by the device 1000 based on control information generated by theserver 3000, according to an exemplary embodiment.

In operation S200, the device 1000 checks the external apparatus 2000controllable by the device 1000. The device 1000 may check the externalapparatus 2000 controllable by the device 1000 from among the externalapparatuses 2000 connectable with the device 1000. The controllableexternal apparatus 2000 may be an external apparatus connected orconnectable with the device 1000 and having installed therein apredetermined application for utilizing a control service according toan exemplary embodiment. If the application is not installed in thecontrollable external apparatus 2000, the device 1000 may performcommunication connection and pairing with the external apparatus 2000,and may provide to the external apparatus 2000 the application or linkinformation for downloading the application.

The external apparatus 2000 may include, for example, a home appliancesuch as a smart TV or an air conditioner, a security camera, a recorder,a mobile phone, a PC, a pillow vibrator, a motor vehicle, or anavigator, but is not limited thereto. The device 1000 and the externalapparatus 2000 may be connected with each other via a wired network suchas a LAN, a WAN, or a VAN, or a wireless network such as a mobile radiocommunication network, an NFC network, or a satellite communicationnetwork. The device 1000 may be connected with the external apparatus2000 via, for example, a home gateway (not shown).

The device 1000 may receive from the checked external apparatus 2000apparatus information of the external apparatus 2000, but is not limitedthereto. The device 1000 may receive the apparatus information of theexternal apparatus 2000 from a separate server (not shown). For example,if the device 1000 is connected with the external apparatus 2000 via ahome gateway (not shown), the device 1000 may request the home gatewayfor the apparatus information of the external apparatus 2000. Theapparatus information of the external apparatus 2000 may includeinformation about at least one of an identification value of theexternal apparatus 2000, a media access control (MAC) address, a serviceset identifier (SSID), the type of the external apparatus 2000,capability provided by the external apparatus 2000, a category, and acommand used to control the external apparatus 2000. The capabilityprovided by the external apparatus 2000 may include, for example, avoice output capability, a video output capability, a voice recordingcapability, an image capturing capability, and a humidity sensingcapability, but is not limited thereto.

After user input information is generated in operation S202 or a user'sintention is determined in operation S204, the device 1000 may searchfor the external apparatus 2000 to be controlled by the device 1000,based on the user input information or the user's intention.

In operation S202, the device 1000 provides the user input informationto the server 3000. The device 1000 may generate the user inputinformation based on a user input, and may provide the generated inputinformation to the server 3000. The user input information may beinformation about at least one of text input by the user, a voice of theuser, a facial expression of the user, a gesture of the user, and aphysical state of the user.

If the user inputs text to the device 1000, the device 1000 may providethe text data input by the user as the user input information to theserver 3000. For example, if the user inputs text such as “Wake me up at7:00 if it doesn't rain tomorrow” to the device 1000, the device 1000may provide the input text data as the user input information to theserver 3000. Otherwise, if the user inputs a voice to the device 1000,the device 1000 may provide the input voice data as the user inputinformation to the server 3000. The device 1000 may transform the inputvoice data into text data, and may transmit the transformed text data asthe user input information to the server 3000. Alternatively, forexample, the device 1000 may generate the user input information byusing an email or a text message stored in the device 1000.

The device 1000 may parse the user input information, and may transmitthe parsed data to the server 3000. If the user input information istext data, the device 1000 may parse the text data. Otherwise, if theuser input information is voice data, the device 1000 may transform thevoice data into text data and may parse the transformed text data.However, an exemplary embodiment is not limited thereto. The device 1000may provide a voice data file to the server 3000, and the server 3000may transform and parse the received voice data file into text.

In operation S204, the server 3000 determines the user's intention basedon the user input information. The server 3000 may determine the user'sintention by analyzing the user input information. In order to analyzereceived text data as the user input information, the server 3000 mayuse various natural language analysis methods. For example, the server3000 may analyze the text data by using natural language processing suchas morpheme analysis, syntax analysis, or named entity recognition.Alternatively, the server 3000 may determine the user's intention basedon the user input information by using, for example, ontology-basedreasoning and probability-based reasoning. If the server 3000 receivesvoice data as the user input information, the server 3000 may transformthe received voice data into text data, and may analyze the transformedtext data. The server 3000 may generate user intention information byanalyzing the user input information. For example, if the user inputinformation is “Wake me up at 7:00 if it doesn't rain tomorrow”, theserver 3000 may generate the user intention information including timeinformation such as “tomorrow, 7:00”, information related to conditioninformation such as “if it doesn't rain”, and operation information suchas “wake me up”. The information related to condition information may beinformation used by the device 1000 or the server 3000 to obtaincondition information about the user's intention.

Alternatively, the server 3000 may determine the user's intention via aninteractive interface using feedback. For example, if the user'sintention is not easily determined based on the user input informationreceived from the device 1000, the server 3000 may request the device1000 for additional user input information. The server 3000 may notifyadditionally needed information to the device 1000. The server 3000 maydetermine the user's intention by using the user input informationadditionally received from the device 1000. For example, if user inputinformation such as “Give me a call if the baby wakes up” is receivedfrom the device 1000, the server 3000 may request the device 1000 toadditionally provide information about a point of time when the babywakes up. The device 1000 may additionally provide user inputinformation such as “Give me a call if the baby wakes up tomorrowmorning” to the server 3000, and the server 3000 may determine theuser's intention based on the additionally received user inputinformation.

In operation S206, the device 1000 provides the apparatus information ofthe checked external apparatus 2000 to the server 3000. The device 1000may provide information about at least one of an identification value ofthe external apparatus 2000, a MAC address, an SSID, the type of theexternal apparatus 2000, capability provided by the external apparatus2000, a category, and a command used to control the external apparatus2000 to the server 3000. In operation S206, the device 1000 may providethe apparatus information of the device 1000 to the server 3000. Theapparatus information of the device 1000 may include information aboutat least one of, for example, an identification value of the device1000, a MAC address, an SSID, the type of the device 1000, capabilityprovided by the device 1000, a category, and a command used to controlthe device 1000.

In operation S208, the server 3000 determines a condition forcontrolling the external apparatus 2000 based on the user's intention.The condition for controlling the external apparatus 2000 is a conditionfor allowing a user-desired operation to be performed according to theuser's intention. For example, if the user input information is “Wake meup if it doesn't rain at 7:00 tomorrow”, a condition such as “if itdoesn't rain at 7:00 tomorrow” for a user-desired operation such as“wake me up” may be the condition for controlling the external apparatus2000. The server 3000 may determine a condition that has to be satisfiedto control operation of the external apparatus 2000 or the device 1000,based on the user intention information. For example, if the userintention information includes time information such as “tomorrow, 7:00”and information related to condition information such as “if it doesn'train”, the server 3000 may determine a time condition such as“tomorrow”, a weather condition such as “sunny, cloudy”, and a placecondition such as “device location” as the condition for controlling theexternal apparatus 2000. In order to ensure that the determinedcondition is satisfied, a predetermined external apparatus 2000 may becontrolled.

In operation S210, the server 3000 selects the external apparatus 2000related to the user's intention. The server 3000 may select the externalapparatus 2000 related to the user's intention, based on operationinformation included in the user intention information and the apparatusinformation received from the device 1000. For example, if the userintention information includes operation information such as “wake meup”, the server 3000 may select the external apparatus 2000 to wake upthe user, by using the apparatus information received from the device1000. The server 3000 may select, for example, a pillow vibrator and/ora lighting device as the external apparatuses 2000 for waking up theuser.

In operation S212, the server 3000 generates control information forsatisfying the user's intention. The control information may includeinformation for checking whether the condition determined by the server3000 is satisfied, and information for controlling operation of at leastone of the device 1000 and the selected external apparatus 2000 if thecondition determined by the server 3000 is satisfied. The controlinformation may include a control command for controlling operation ofthe device 1000 or the external apparatus 2000, and information togenerate the control command.

In more detail, the server 3000 may generate the control information forchecking whether the condition determined by the server 3000 issatisfied, based on the condition determined by the server 3000 and theapparatus information received from the device 1000. For example, if thecondition determined by the server 3000 includes a time condition suchas “tomorrow”, a weather condition such as “sunny, cloudy”, and a placecondition such as “device location”, the server 3000 may generate thecontrol information for checking whether the condition determined by theserver 3000 is satisfied by allowing the device 1000 to check the date,the location of the device 1000, and the weather via the device 1000 orthe external apparatus 2000. The control information may include acontrol command instructing to check the date and the weather, a controlcommand instructing to obtain humidity information from a humiditysensor, and a command instructing to check the location of the device1000, and the control commands included in the control information maybe aligned in a predetermined order.

However, the control information is not limited thereto and may omit acontrol command but may include information referred to so as togenerate a control command, for example, “determine whether the weatheris sunny or cloudy at a device location tomorrow”.

If the condition determined by the server 3000 is satisfied, the server3000 may generate the control information for operating the device 1000or the external apparatus 2000. The server 3000 may generate the controlinformation for operating the device 1000 or the external apparatus2000, based on the operation information included in the user intentioninformation and the apparatus information received from the device 1000.For example, if the user intention information includes operationinformation such as “wake me up”, the server 3000 may generate a controlcommand instructing the device 1000 to output alarm information, acontrol command instructing a lighting device to be turned on, and/or acontrol command instructing a pillow vibrator to generate vibration.

The control information may include a plurality of control commands forcontrolling operation of at least one of the device 1000 and theexternal apparatus 2000, and the plurality of control commands may beexecuted in a predetermined order by at least one of the device 1000 andthe external apparatus 2000.

In operation S214, the server 3000 provides the generated controlinformation to the device 1000.

In operation S216, the device 1000 checks whether the condition forcontrolling the external apparatus 2000 is satisfied. The device 1000may check whether the condition determined by the server 3000 issatisfied, based on the control information received from the server3000. If the control information received from the server 3000 includesa control command for checking whether the condition for controlling theexternal apparatus 2000 is satisfied, the device 1000 may operateaccording to the control command included in the control information tocheck whether the condition is satisfied. Otherwise, if the controlinformation received from the server 3000 does not include the controlcommand for checking whether the condition for controlling the externalapparatus 2000 is satisfied, the device 1000 may generate a controlcommand based on the control information, and may check whether thecondition is satisfied, according to the generated control command.

In operation S218, the device 1000 provides the control command to theexternal apparatus 2000 based on the control information. The device1000 may transmit the control command to the external apparatus 2000 inorder to follow the user's intention if the condition for controllingthe external apparatus 2000 is satisfied. If the condition forcontrolling the external apparatus 2000 is satisfied, the device 1000may transmit the control commands for operating the external apparatuses2000 to the external apparatuses 2000 in a predetermined order. Thedevice 1000 may perform a predetermined operation based on the controlinformation.

In operation S220, another external apparatus (not shown) isadditionally connected with the device 1000 and, in operation S224, thedevice 1000 provides apparatus information of the additionally connectedexternal apparatus to the server 3000.

In operation S226, the server 3000 generates control informationincluding a control command about the additionally connected externalapparatus. The server 3000 may change the control information forsatisfying the user's intention, based on the apparatus information ofthe additionally connected external apparatus.

In operation S228, the server 3000 provides the changed controlinformation to the device 1000. As such, the device 1000 may controloperation of the device 1000 or the external apparatus 2000 based on thechanged control information.

FIG. 3 is a flowchart of a method of controlling the external apparatus2000 by the device 1000 based on control information generated by theserver 3000, according to an exemplary embodiment. In FIG. 3, the server3000 may check whether a condition for controlling the externalapparatus 2000 is satisfied. Since some of the operations of FIG. 3 aresimilar to those described above with reference to FIG. 2, the detaileddescription thereof is omitted.

In operation S300, the device 1000 checks the external apparatus 2000controllable by the device 1000.

In operation S302, the device 1000 provides user input information andthe apparatus information to the server 3000. The device 1000 maygenerate the user input information based on a user input, and mayprovide the generated input information to the server 3000.

Also, the device 1000 may provide information about at least one of anidentification value of the external apparatus 2000, the type of theexternal apparatus 2000, and a command used to control the externalapparatus 2000 to the server 3000.

In operation S304, the server 3000 determines the user's intention basedon the user input information. The server 3000 may determine the user'sintention by analyzing the user input information. In order to analyzereceived text data as the user input information, the server 3000 mayuse various natural language analysis methods. For example, the server3000 may analyze the text data by using natural language processing suchas morpheme analysis, syntax analysis, or named entity recognition. Ifthe server 3000 receives voice data as the user input information, theserver 3000 may transform the received voice data into text data, andmay analyze the transformed text data. Also, the server 3000 maygenerate user intention information by analyzing the user inputinformation.

In operation S306, the server 3000 determines a condition forcontrolling the external apparatus 2000 based on the user's intention.The server 3000 may determine a condition that has to be satisfied tocontrol operation of the external apparatus 2000 or the device 1000,based on the user intention information. For example, if the userintention information includes time information such as “tomorrow, 7:00”and information related to condition information such as “if it doesn'train”, the server 3000 may determine a time condition such as“tomorrow”, a weather condition such as “sunny, cloudy”, and a placecondition such as “device location” as the condition for controlling theexternal apparatus 2000.

In operation S308, the server 3000 requests the device 1000 forcondition information. The server 3000 may request the device 1000 forthe condition information obtainable by the device 1000 and thecondition information obtainable by the external apparatus 2000, basedon the condition information. The server 3000 may request the device1000 for the condition information needed to check whether thedetermined condition is satisfied. For example, if a time condition, aweather condition, and a place condition are determined, the server 3000may request the device 1000 to provide information about the weather ata fixed time at a fixed place as the condition information to the server3000. The server 3000 may provide the type of the condition informationto the device 1000, but is not limited thereto. The server 3000 mayprovide to the device 1000 a control command instructing the device 1000and the external apparatus 2000 to obtain the condition information.

In operation S310, the device 1000 obtains the condition information.The device 1000 may obtain the condition information in response to arequest to obtain the condition information. The device 1000 may obtainvarious types of the condition information from the external apparatus2000 and a separate server (not shown). For example, the device 1000 mayobtain location information of the device 1000 from a locationinformation providing server (not shown), may obtain weather informationaround the device 1000 from a weather server (not shown), and may obtaintraffic information about a predetermined route from a trafficinformation providing server (not shown).

Also, for example, the device 1000 may obtain humidity information froma humidity sensor (not shown), may obtain a captured image from aphotographing apparatus (not shown), and may obtain recorded voice datafrom a recording apparatus (not shown). The device 1000 may obtain thecondition information from an apparatus included in the device 1000. Forexample, the device 1000 may obtain various types of the conditioninformation from a temperature sensor, a humidity sensor, a geomagneticsensor, a gravity sensor, a motion sensor, a gyro sensor, a camera, anda recorder included in the device 1000. The device 1000 may obtain thecondition information about a physical state of the user. The conditioninformation about the physical state of the user may be, for example,information about a heart rate, an electrocardiogram (ECG), breathing, apulse, and physical exercise of the user, and may be obtained from thedevice 1000 or the external apparatus 2000 connected with the device1000. Furthermore, the device 1000 may obtain the condition informationabout a network status, a login status of the external apparatus 2000,an application installed in the device 1000 or the external apparatus2000, and a user using the application. However, the device 1000 is notlimited thereto and may obtain various types of information as thecondition information.

In operation S312, the device 1000 provides the obtained conditioninformation to the server 3000. The device 1000 may obtain the conditioninformation in a preset cycle, and may provide the obtained conditioninformation to the server 3000 in real time, but is not limited thereto.The server 3000 may obtain the condition information from another device(not shown) or another server (not shown).

In operation S314, the server 3000 checks whether the condition forcontrolling the external apparatus 2000 is satisfied, based on thecondition information. The server 3000 may check whether the conditionis satisfied, based on the condition information received from at leastone of the device 1000, the other device, and the other server.

In operation S316, the server 3000 selects the external apparatus 2000related to the user's intention. The server 3000 may check an operationfor satisfying the user's intention, and may select the externalapparatus 2000 for performing the checked operation. For example, if theuser intention information includes operation information such as “wakeme up”, the server 3000 may select a lighting device and/or a pillowvibrator.

In operation S318, the server 3000 generates control information aboutthe selected external apparatus 2000. If the condition determined by theserver 3000 is satisfied, the server 3000 may generate the controlinformation for operating the device 1000 or the external apparatus2000. The server 3000 may generate the control information for operatingthe device 1000 or the external apparatus 2000, based on the operationinformation included in the user intention information and the apparatusinformation received from the device 1000. For example, if the userintention information includes operation information such as “wake meup”, the server 3000 may generate a control command instructing thedevice 1000 to output alarm information, a control command instructing alighting device to be turned on, and a control command instructing apillow vibrator to generate vibration. The control commands included inthe control information generated by the server 3000 may be aligned in apredetermined order.

In operation S320, the server 3000 provides the generated controlinformation to the device 1000.

In operation S322, the device 1000 transmits the control command to theexternal apparatus 2000 based on the control information. The device1000 may transmit the control commands for operating the externalapparatuses 2000 to the external apparatuses 2000 in a predeterminedorder. The device 1000 may perform a predetermined operation based onthe control information.

In operation S324, another external apparatus (not shown) isadditionally connected with the device 1000 and, in operation S326, thedevice 1000 provides apparatus information of the additionally connectedexternal apparatus to the server 3000.

In operation S328, the server 3000 generates control informationincluding a control command about the additionally connected externalapparatus. The server 3000 may change the control information forsatisfying the user's intention, based on the apparatus information ofthe additionally connected external apparatus.

In operation S330, the server 3000 provides the changed controlinformation to the device 1000. As such, the device 1000 may controloperation of the device 1000 or the external apparatus 2000 based on thechanged control information.

FIG. 4 is a flowchart of a method of generating a control command andcontrolling the external apparatus 2000 by the device 1000 based on userintention information received from the server 3000, according to anexemplary embodiment. In FIG. 4, the device 1000 may determine acondition for controlling the external apparatus 2000, based on the userintention information. Since some of the operations of FIG. 4 aresimilar to those described above with reference to FIGS. 2 and 3, thedetailed description thereof is omitted.

In operation S400, the device 1000 checks the external apparatus 2000controllable by the device 1000 and, in operation S402, the device 1000provides user input information to the server 3000. In operation S404,the server 3000 determines a user's intention based on the user inputinformation. Operations S400, S402, and S404 may respectively correspondto operations S200, S202, and S204.

In operation S406, the server 3000 provides user intention informationto the device 1000. For example, if the user input information is “Wakeme up at 7:00 if it doesn't rain tomorrow”, the server 3000 may provideto the device 1000 the user intention information including timeinformation such as “tomorrow, 7:00”, condition information such as “ifit doesn't rain”, and operation information such as “wake me up”.

In operation S408, the device 1000 determines a condition forcontrolling the external apparatus 2000 based on the user's intention.The device 1000 may decide a condition that has to be satisfied tocontrol operation of the external apparatus 2000 or the device 1000,based on the user intention information. For example, if the userintention information includes time information such as “tomorrow, 7:00”and condition information such as “if it doesn't rain”, the device 1000may decide a time condition such as “tomorrow”, a weather condition suchas “sunny, cloudy”, and a place condition such as “device location” asthe condition for controlling the external apparatus 2000.

In operation S410, the device 1000 obtains condition information. Thedevice 1000 may obtain various types of the condition information fromthe external apparatus 2000 and a separate server (not shown).

In operation S412, the device 1000 obtains apparatus information of theexternal apparatus 2000. The device 1000 may receive the apparatusinformation of the external apparatus 2000 from the external apparatus2000, but is not limited thereto. The device 1000 may receive theapparatus information of the external apparatus 2000 from a separateserver (not shown). For example, if the device 1000 is connected withthe external apparatus 2000 via a home gateway (not shown), the device1000 may request the home gateway for the apparatus information of theexternal apparatus 2000. The apparatus information of the externalapparatus 2000 may include information about at least one of anidentification value of the external apparatus 2000, the type of theexternal apparatus 2000, and a command used to control the externalapparatus 2000.

In operation S414, the device 1000 selects the external apparatus 2000related to the user's intention. The device 1000 may check an operationfor satisfying the user's intention, and may select the externalapparatus 2000 for performing the checked operation.

In operation S416, the device 1000 generates control information aboutthe selected external apparatus 2000. If the condition determined by thedevice 1000 is satisfied, the device 1000 may generate the controlinformation for operating the device 1000 or the external apparatus2000. The device 1000 may generate the control information for operatingthe device 1000 or the external apparatus 2000, based on the operationinformation included in the user intention information and the apparatusinformation received from the device 1000. The control information mayinclude control commands about at least one external apparatus 2000 andthe device 1000, and the control commands may be aligned in apredetermined order.

In operation S418, the device 1000 provides a predetermined controlcommand to the external apparatus 2000 based on the generated controlinformation.

In operation S420, another external apparatus (not shown) isadditionally connected with the device 1000 and, in operation S422, thedevice 1000 generates control information including a control commandabout the additionally connected external apparatus. The device 1000 maychange the control information for satisfying the user's intention,based on apparatus information of the additionally connected externalapparatus. As such, the device 1000 may control operation of the device1000 or the external apparatus 2000 based on the changed controlinformation.

FIG. 5 is a flowchart of a method of directly determining a user'sintention and controlling the external apparatus 2000 by the device 1000based on user input information, according to an exemplary embodiment.

In operation S500, the device 1000 checks the external apparatus 2000controllable by the device 1000.

In operation S502, the device 1000 determines a user's intention basedon user input information. The device 1000 may determine the user'sintention by analyzing the user input information. In order to analyzethe user input information, the device 1000 may use various naturallanguage analysis methods. For example, the device 1000 may analyze textdata as the user input information by using natural language processingsuch as morpheme analysis, syntax analysis, or named entity recognition.If the user input information is voice data, the device 1000 maytransform the voice data into text data, and may analyze the transformedtext data. The device 1000 may generate user intention information byanalyzing the user input information. For example, if the user inputinformation is “Wake me up at 7:00 if it doesn't rain tomorrow”, thedevice 1000 may generate the user intention information including timeinformation such as “tomorrow, 7:00”, condition information such as “ifit doesn't rain”, and operation information such as “wake me up”.

In operation S504, the device 1000 determines a condition forcontrolling the external apparatus 2000 based on the user's intentionand, in operation S506, the device 1000 obtains condition information.In operation S508, the device 1000 obtains apparatus information of theexternal apparatus 2000, in operation S510, the device 1000 selects theexternal apparatus 2000 related to the user's intention and, inoperation S512, the device 1000 generates control information about theselected external apparatus 2000. In operation S514, the device 1000provides a predetermined control command to the external apparatus 2000based on the generated control information.

Operations S504, S506, S508, S510, S512, and S514 may correspond to theoperations described above with reference to FIGS. 2, 3, and 4, andthus, are not described in detail again.

FIG. 6 is an operation table of the device 1000 for controlling theexternal apparatus 2000, according to an exemplary embodiment.Information about operation of the device 1000 in relation to user inputinformation is recorded in the operation table of the device 1000.

As illustrated in FIG. 6, the operation table 58 of the device 1000 mayinclude a user input information field 60, a user's intention field 62,an external apparatus controlling condition field 64, and an operationfield 66. The operation table of the device 1000 may be generated by atleast one of the device 1000 and the server 3000.

User input information obtained by the device 1000 is recorded in theuser input information field 60. If a user inputs text to the device1000, the text data input by the user may be recorded in the user inputinformation field 60. If the user input information is voice data, thevoice data may be transformed into text data, and the transformed textdata may be stored in the user input information field 60.

The user input information recorded in the user input information field60 may include, for example, “Wake me up at 7:00 if it doesn't raintomorrow”, “Let's go home if the traffic is not heavy before 20:00tomorrow”, and “Give me a video call if the baby wakes up”.

Information about a user's intention determined based on the user inputinformation may be recorded in the user's intention field 62. The userinput information may be analyzed by using natural language processingsuch as morpheme analysis, syntax analysis, or named entity recognition.For example, if the user input information is “Wake me up at 7:00 if itdoesn't rain tomorrow”, “When: tomorrow, 7:00, Condition: if it doesn'train, Do: wake me up” may be recorded in the user's intention field 62.

Information about a condition that has to be satisfied to control theexternal apparatus 2000 according to the user's intention may berecorded in the external apparatus controlling condition field 64.

For example, based on the information recorded in the user's intentionfield 62, the external apparatus controlling condition field 64 maycontain “Who: x, What: weather, When: tomorrow, Condition: sunny,cloudy, Where: device location”. That is, a subject condition such as“weather”, a time condition such as “tomorrow”, a weather condition suchas “sunny, cloudy”, and a place condition such as “device location” maybe recorded in the external apparatus controlling condition field 64. Inaddition to “Who”, “What”, “When”, “Condition”, and “Where”, variouscondition items may be further included in the external apparatuscontrolling condition field 64. If predetermined data is not recorded asa condition item, the server 3000 may request the device 1000 foradditional user input information about the condition item having norecorded data, and may receive the additional user input informationfrom the device 1000 to record predetermined data as the condition item.That is, if the user's intention is not determined based on the userinput information, or if a condition and operation for controlling theexternal apparatus 2000 according to the user's intention are notdetermined, the server 3000 may request the device 1000 for additionaluser input information.

The device 1000 may generate a control command for checking whether apredetermined condition is satisfied, based on the information recordedin the condition field 64.

Operation information of the device 1000 and the external apparatus 2000for determining whether the condition for controlling the externalapparatus 2000 is satisfied, and for operating the device 1000 and theexternal apparatus 2000 according to the user's intention may berecorded in the operation field 66.

For example, based on the information recorded in the user's intentionfield 62, “check time”, “check device location”, and “check weather” maybe recorded in the operation field 66. If the condition for controllingthe external apparatus 2000 is satisfied, as an operation of the device1000 and the external apparatus 2000, “output alarm information” may berecorded in the operation field 66.

Based on the operation information recorded in the operation field 66,control information of the device 1000 and the external apparatus 2000for performing the operation recorded in the operation field 66 may begenerated. The control information may include a plurality of controlcommands for controlling operation of at least one of the device 1000and the external apparatus 2000, and the plurality of control commandsmay be executed by at least one of the device 1000 and the externalapparatus 2000 in a predetermined order. For example, based on “checktime”, “check device location” and “check weather” recorded in theoperation field 66, a control command may be generated in such a waythat the device 1000 checks the time, the location of the device 1000,and the weather by using at least one of the device 1000, a separateserver (not shown), and the external apparatus 2000. For example, basedon “output alarm information” recorded in the operation field 66, acontrol command may be generated in such a way that the device 1000 andthe external apparatus 2000 output predetermined alarm information.

Although the device 1000 determines the user's intention and determinesthe condition and operation for controlling the external apparatus 2000in the above description, exemplary embodiments are not limited thereto.The determining of the condition and operation for controlling theexternal apparatus 2000 based on the user input information by thedevice 1000 may be included in the determining of the user's intention.

FIG. 7 is a schematic diagram showing that the external apparatus 2000is controlled according to user input information that is input to thedevice 1000, in a system for controlling the external apparatus 2000,according to an exemplary embodiment.

Referring to FIG. 7, if a user inputs voice data such as “Wake me up at7:00 if it doesn't rain tomorrow” to the device 1000, the device 1000may provide the voice data or text data transformed from the voice data,as user input information to the server 3000. The device 1000 mayprovide apparatus information of a humidity sensor 72, a lighting device74, and a pillow vibrator 76 connectable with the device 1000, to theserver 3000.

The server 3000 may determine the user's intention based on the userinput information, and may determine a condition for controlling theexternal apparatus 2000. The server 3000 may determine whether thedetermined condition is satisfied, may generate control information forcontrolling the device 1000 and the external apparatus 2000, and mayprovide the generated control information to the device 1000.

The device 1000 may determine whether the condition is satisfied, byobtaining weather information from a weather server 70 based on thereceived control information, and obtaining humidity information fromthe humidity sensor 72. If the condition is satisfied, the device 1000may transmit a control command instructing to power on to the lightingdevice 74 and/or may transmit a control command instructing to generatevibration to the pillow vibrator 76.

FIG. 8 is a schematic diagram showing that the external apparatus 2000is controlled according to user input information that is input to thedevice 1000, in a system for controlling the external apparatus 2000,according to an exemplary embodiment.

Referring to FIG. 8, if a user inputs voice data such as “Let's go homeif the traffic is not heavy before 21:00 tomorrow” to the device 1000,the device 1000 may provide the voice data or text data transformed fromthe voice data, as user input information to the server 3000. The device1000 may provide apparatus information of a motor vehicle 82 and anavigator 84 connectable with the device 1000, to the server 3000.

The server 3000 may determine the user's intention based on the userinput information, and may determine a condition for controlling theexternal apparatus 2000. The server 3000 may determine whether thedetermined condition is satisfied, may generate control information forcontrolling the device 1000 and the external apparatus 2000, and mayprovide the generated control information to the device 1000.

The device 1000 may determine whether the condition is satisfied, byobtaining traffic information about a route from a current location ofthe device 1000 to home, from a traffic information providing server 80based on the received control information. If the condition issatisfied, the device 1000 may transmit a control command instructing tostart an engine to the motor vehicle 82 and may transmit a controlcommand instructing to output route guidance information to thenavigator 84.

FIG. 9 is a schematic diagram showing that the external apparatus 2000is controlled according to user input information that is input to thedevice 1000, in a system for controlling the external apparatus 2000,according to an exemplary embodiment.

Referring to FIG. 9, if a user inputs voice data such as “Give me avideo call if the baby wakes up” to the device 1000, the device 1000 mayprovide the voice data or text data transformed from the voice data, asuser input information to the server 3000. The device 1000 may provideapparatus information of a photographing apparatus 90, a recordingapparatus 92, and a telephone 94 connectable with the device 1000, tothe server 3000.

The server 3000 may determine the user's intention based on the userinput information, and may determine a condition for controlling theexternal apparatus 2000. The server 3000 may determine whether thedetermined condition is satisfied, may generate control information forcontrolling the device 1000 and the external apparatus 2000, and mayprovide the generated control information to the device 1000.

The device 1000 may determine whether the condition is satisfied, byobtaining a captured image from the photographing apparatus 90 andobtaining recorded voice data from the recording apparatus 92 based onthe received control information. If the condition is satisfied, thedevice 1000 may transmit a control command instructing to make a videocall to the telephone 94.

FIG. 10 is a block diagram of the device 1000 according to an exemplaryembodiment.

As illustrated in FIG. 10, the device 1000 includes an externalapparatus checker 1100, an input unit 1200, an information provider1300, a condition information obtainer 1400, a control informationobtainer 1500, a control command provider 1600, a database (DB) 1700, atransceiver 1800, and a controller 1900.

The external apparatus checker 1100 checks the external apparatus 2000controllable by the device 1000. The external apparatus checker 1100 maycheck the external apparatus 2000 controllable by the device 1000 fromamong the external apparatuses connectable with the device 1000. Forexample, the external apparatus checker 1100 may identify or select theexternal apparatus 2000 based on criteria, such as connectivity,compatibility, ability to share functionality, etc., with the device1000.

The external apparatus checker 1100 may receive from the checkedexternal apparatus 2000 apparatus information of the external apparatus2000, but is not limited thereto. The external apparatus checker 1100may receive the apparatus information of the external apparatus 2000from a separate server (not shown). For example, if the device 1000 isconnected with the external apparatus 2000 via a home gateway (notshown), the external apparatus checker 1100 may request the home gatewayfor the apparatus information of the external apparatus 2000. Theapparatus information of the external apparatus 2000 may includeinformation about at least one of an identification value of theexternal apparatus 2000, the type of the external apparatus 2000, and acommand used to control the external apparatus 2000.

The input unit 1200 receives a user input about the device 1000, andgenerates user input information. The user input information may includetext data or voice data, but is not limited thereto. Alternatively, theinput unit 1200 may generate the user input information by using anemail or a text message stored in the device 1000.

The information provider 1300 provides to the server 3000 various typesof information for generating control information. The informationprovider 1300 may provide to the server 3000 at least one of theapparatus information of the external apparatus 2000, the user inputinformation, and condition information.

The condition information obtainer 1400 may obtain the conditioninformation from at least one of the device 1000, a separate server (notshown), and the external apparatus 2000. The condition informationobtainer 1400 may obtain the condition information based on the controlinformation obtained by the control information obtainer 1500 to bedescribed below, but is not limited thereto. In order to satisfy auser's intention, the condition information obtainer 1400 may obtain thecondition information related to a condition for controlling theexternal apparatus 2000.

The control information obtainer 1500 obtains the control informationfor controlling the device 1000 and the external apparatus 2000according to the user's intention. The control information obtainer 1500may receive the control information generated by the server 3000, but isnot limited thereto. The control information obtainer 1500 may determinethe user's intention based on the user input information, may determinethe condition for controlling the external apparatus 2000, and maygenerate the control information. The control information obtainer 1500may receive information about at least one of the condition forcontrolling the external apparatus 2000 and the user's intention fromthe server 3000, and may generate the control information. The controlinformation obtainer 1500 will be described in detail below withreference to FIG. 11.

The control command provider 1600 provides a predetermined controlcommand to the external apparatus 2000 based on the obtained controlinformation. The control information may include a plurality of controlcommands aligned in a predetermined order, and the control commandprovider 1600 may provide the plurality of control commands to at leastone external apparatuses 2000 in a predetermined order.

The DB 1700 stores various types of information to control the device1000 and the external apparatus 2000 by the device 1000 according to theuser's intention.

The transceiver 1800 transmits and receives to and from the externalapparatus 2000 and the server 3000 the various types of informationneeded to control the device 1000 and the external apparatus 2000 by thedevice 1000 according to the user's intention.

The controller 1900 controls operations of the device 1000, and controlsthe external apparatus checker 1100, the input unit 1200, theinformation provider 1300, the condition information obtainer 1400, thecontrol information obtainer 1500, the control command provider 1600,the DB 1700, and the transceiver 1800 in such a way that the device 1000controls the device 1000 and the external apparatus 2000 according tothe user's intention.

Some or all of the external apparatus checker 1100, the input unit 1200,the information provider 1300, the condition information obtainer 1400,the control information obtainer 1500, and the control command provider1600 may be driven by software modules, but this is not limiting. Someor all of the external apparatus checker 1100, the input unit 1200, theinformation provider 1300, the condition information obtainer 1400, thecontrol information obtainer 1500, and the control command provider 1600may be hardware devices.

Also, at least some of the external apparatus checker 1100, the inputunit 1200, the information provider 1300, the condition informationobtainer 1400, the control information obtainer 1500, and the controlcommand provider 1600 may be included in the controller 1900, and theexternal apparatus checker 1100, the input unit 1200, the informationprovider 1300, the condition information obtainer 1400, the controlinformation obtainer 1500, the control command provider 1600, and thecontroller 1900 may be driven by one processor. However, an exemplaryembodiment is not limited thereto.

FIG. 11 is a block diagram of the condition information obtainer 1400according to an exemplary embodiment.

As illustrated in FIG. 11, the condition information obtainer 1400 mayinclude a user intention determiner 1410, an apparatus control conditiondeterminer 1420, a control information generator 1430, and a controlinformation receiver 1440.

The user intention determiner 1410 determines a user's intention byanalyzing the user input information. For example, the user intentiondeterminer 1410 may use various natural language analysis methods. Forexample, the device 1000 may analyze text data as the user inputinformation by using natural language processing such as morphemeanalysis, syntax analysis, or named entity recognition. If the userinput information is voice data, the user intention determiner 1410 maytransform the voice data into text data, and may analyze the transformedtext data. The user intention determiner 1410 may generate userintention information by analyzing the user input information. Forexample, if the user input information is “Wake me up at 7:00 if itdoesn't rain tomorrow”, the user intention determiner 1410 may generatethe user intention information including time information such as“tomorrow, 7:00”, condition information such as “if it doesn't rain”,and operation information such as “wake me up”. Alternatively, the userintention determiner 1410 may receive from the server 3000 the userintention information generated by the server 3000.

The apparatus control condition determiner 1420 determines a conditionfor controlling the external apparatus 2000 based on the user'sintention. The apparatus control condition determiner 1420 may determinea condition that has to be satisfied to control operation of theexternal apparatus 2000 or the device 1000, based on the user intentioninformation. For example, if the user intention information includestime information such as “tomorrow, 7:00” and condition information suchas “if it doesn't rain”, the apparatus control condition determiner 1420may determine a time condition such as “tomorrow”, a weather conditionsuch as “sunny, cloudy”, and a place condition such as “device location”as the condition for controlling the external apparatus 2000.Alternatively, the apparatus control condition determiner 1420 mayreceive from the server 3000 information about the condition determinedthe server 3000.

The control information generator 1430 generates control informationabout the device 1000 and the external apparatus 2000. The controlinformation generator 1430 may determine whether the condition issatisfied, based on the condition information. If the conditiondetermined by the device 1000 is satisfied, the control informationgenerator 1430 may generate the control information for operating thedevice 1000 or the external apparatus 2000. The control informationgenerator 1430 may generate the control information for operating thedevice 1000 or the external apparatus 2000, based on operationinformation, and apparatus information of the external apparatus 2000,which are included in the user intention information. The controlinformation may include control commands about at least one of theexternal apparatus 2000 and the device 1000, and the control commandsmay be aligned in a predetermined order.

The control information receiver 1440 may receive from the server 3000the control information generated by the server 3000.

FIG. 12 is a block diagram of the device 1000 according to an exemplaryembodiment.

A mobile communicator 1001 performs, for example, call setup and datacommunication with a base station via a cellular network such as a 3G/4Gnetwork. A sub communicator 1002 performs a function for short-distancecommunication such as Bluetooth or NFC. A broadcasting receiver 1003receives a digital multimedia broadcasting (DMB) signal.

A camera 1004 includes a lens and optical elements for capturing a photoor video.

A sensor 1005 may include a gravity sensor for sensing motion of thedevice 1000, an illumination sensor for sensing brightness of light, aproximity sensor for sensing proximity of a person, a motion sensor forsensing motion of a person, etc.

A GPS receiver 1006 receives a GPS signal from a satellite. Variousservices may be provided to a user by using GPS signals.

An input/output (I/O) unit 1010 provides an interface with the externalapparatus 2000 or the user, and includes buttons 1011, a microphone1012, a speaker 1013, a vibration motor 1014, a connector 1015, and/or akeypad 1016.

A touch screen 1018 receives a touch input of the user. A touch screencontroller 1017 transmits to a controller 1050 the touch input receivedby the touch screen 1018. A power supply source 1019 is connected with abattery or an external power source to supply power to the device 1000.

The controller 1050 provides a control command to the external apparatus2000 by executing programs stored in a memory 1060.

The programs stored in the memory 1060 may be classified into aplurality of modules according to their functions, for example, a mobilecommunication module 1061, a Wi-Fi module 1062, a Bluetooth module 1063,a DMB module 1064, a camera module 1065, a sensor module 1066, a GPSmodule 1067, a video player module 1068, an audio player module 1069, apower module 1070, a touch screen module 1071, a user interface (UI)module 1072, and an application module 1073.

Functions of most of the modules would be instinctively understood byone of ordinary skill in the art in view of their names and thus onlythe application module 1073 will be described here. The applicationmodule 1073 may check the external apparatus 2000 connected with thedevice 1000 by using the mobile communicator 1001 and the subcommunicator 1002, and may receive apparatus information from theexternal apparatus 2000. The application module 1073 may receive a userinput by using the I/O unit 1010, and may generate user inputinformation. Furthermore, the application module 1073 may directlyobtain condition information via the camera 1004, the sensor 1005, orthe GPS receiver 1006, or may receive the condition information from theexternal apparatus 2000 or a separate server (not shown) by using themobile communicator 1001 and the sub communicator 1002. Besides, theapplication module 1073 may obtain control information for controllingthe device 1000 and the external apparatus 2000. The application module1073 may determine the user's intention based on the user inputinformation or may receive user intention information from the server3000. The application module 1073 may determine a condition forcontrolling the external apparatus 2000 based on the user's intention,or may receive information about the condition from the server 3000. Theapplication module 1073 may determine whether the condition is satisfiedand may generate the control information for controlling the device 1000and the external apparatus 2000. The application module 1073 may receivethe control information generated by the server 3000. Furthermore, theapplication module 1073 may provide to the external apparatus 2000 acontrol command included in the control information.

FIG. 13 is a block diagram of the server 3000 according to an exemplaryembodiment.

As illustrated in FIG. 13, the server 3000 includes an input informationreceiver 3100, an apparatus information receiver 3200, a user intentiondeterminer 3300, a condition information receiver 3400, a controlinformation generator 3500, an information provider 3600, a DB 3700, atransceiver 3800, and a controller 3900.

The input information receiver 3100 receives user input information fromthe device 1000. The user input information may include at least one oftext data and voice data, but is not limited thereto.

The apparatus information receiver 3200 receives apparatus informationof the external apparatus 2000 from the device 1000. The apparatusinformation may include information including at least one of anidentification value of the external apparatus 2000, the type of theexternal apparatus 2000, and a command used to control the externalapparatus 2000.

The user intention determiner 3300 determines a user's intention byanalyzing the user input information. For example, the user intentiondeterminer 3300 may use various natural language analysis methods.

The condition information receiver 3400 receives condition informationfrom the device 1000. The condition information receiver 3400 mayrequest the device 1000 for the condition information and may receivefrom the device 1000 the condition information obtained by the device1000.

The control information generator 3500 generates control information forcontrolling the device 1000 and the external apparatus 2000. The controlinformation generator 3500 may generate the control information based onat least one of the user intention information, the apparatusinformation, and the condition information. The control informationgenerator 3500 may determine the user's intention based on the userinput information, may determine the condition for controlling theexternal apparatus 2000, and may generate the control information.

The information provider 3600 may provide to the device 1000 at leastone of the user intention information, and the condition information andthe control information for controlling the external apparatus 2000.

The DB 3700 stores various types of information used when the server3000 generates the control information for operating the device 1000 andthe external apparatus 2000 according to the user's intention based onthe user input information, and provides the generated controlinformation to the device 1000.

The transceiver 3800 transmits to and receives from the device 1000 thevarious types of information used when the server 3000 generates thecontrol information for operating the device 1000 and the externalapparatus 2000 according to the user's intention based on the user inputinformation, and provides the generated control information to thedevice 1000.

The controller 3900 controls operations of the server 3000, and controlsthe input information receiver 3100, the apparatus information receiver3200, the user intention determiner 3300, the condition informationreceiver 3400, the control information generator 3500, the informationprovider 3600, the DB 3700, and the transceiver 3800 in such a way thatthe server 3000 generates the control information for operating thedevice 1000 and the external apparatus 2000 according to the user'sintention based on the user input information, and provides thegenerated control information to the device 1000.

Some or all of the input information receiver 3100, the apparatusinformation receiver 3200, the user intention determiner 3300, thecondition information receiver 3400, the control information generator3500, and the information provider 3600 may be driven by softwaremodules, but are not limited thereto. Some or all of the inputinformation receiver 3100, the apparatus information receiver 3200, theuser intention determiner 3300, the condition information receiver 3400,the control information generator 3500, and the information provider3600 may be hardware devices.

Also, at least some of the input information receiver 3100, theapparatus information receiver 3200, the user intention determiner 3300,the condition information receiver 3400, the control informationgenerator 3500, and the information provider 3600 may be included in thecontroller 3900, and the input information receiver 3100, the apparatusinformation receiver 3200, the user intention determiner 3300, thecondition information receiver 3400, the control information generator3500, the information provider 3600, and the controller 3900 may bedriven by one processor. However, an exemplary embodiment is not limitedthereto.

Exemplary embodiments may be implemented as a medium includingcomputer-executable commands, e.g., a computer-executable programmodule. A computer-readable medium may be an arbitrary medium that maybe accessed by a computer, and may include volatile and nonvolatilemedia, and detachable and non-detachable media. The computer-readablemedium may include a computer recording medium and/or a communicationmedium. The computer recording medium includes volatile and nonvolatilemedia, and detachable and non-detachable media that are embodied byusing an arbitrary method or technology for storing information such asa computer-readable command, a data structure, a program module, orother data. The communication medium may include a computer-readablecommand, a data structure, a program module, or other transmissionmechanisms, and includes an arbitrary information transmission medium.

Exemplary embodiments may, however, be embodied in many different formsand should not be construed as being limited to the embodiments setforth herein; rather, the described exemplary embodiments are providedso that the disclosure will be thorough and complete, and will fullyconvey the inventive concept to those of ordinary skill in the art. Forexample, a single component may be separated into a plurality ofcomponents, while a plurality of components may be combined into onecomponent.

The foregoing exemplary embodiments and advantages are merely exemplaryand are not to be construed as limiting. The present teaching can bereadily applied to other types of apparatuses. The description of theexemplary embodiments is intended to be illustrative, and not to limitthe scope of the claims, and many alternatives, modifications, andvariations will be apparent to those skilled in the art.

What is claimed is:
 1. A method of controlling external devices, themethod comprising: receiving, from a server, first identificationinformation of a plurality of external devices and first capabilityinformation regarding capabilities that the plurality of externaldevices are able to perform; receiving, from an electronic devicedistinct from the plurality of external devices, a single user voicecommand input by a user to the electronic device; obtaining a userintent regarding the received single user voice command based on thereceived single user voice command; identifying at least two externaldevices, from among the plurality of external devices, based on theobtained user intent regarding the single user voice command and thefirst identification information of the plurality of external devices;generating control information for control commands controlling theidentified at least two external devices, based on the obtained userintent regarding the single user voice command, second capabilityinformation of the identified at least two external devices, and secondidentification information of the identified at least two externaldevices; and outputting the generated control information for thecontrol commands controlling the identified at least two externaldevices.
 2. The method of claim 1, wherein: the generated controlinformation comprises capability information for controlling operationsof the identified at least two external devices, and the capabilityinformation is used by the identified at least two external devices in apredetermined order.
 3. The method of claim 1, wherein: the single uservoice command is a phrase which is input into the electronic device by auser voice, and comprises a combinational instruction, and thecombinational instruction is directed to the electronic device andcomprises an action, a time, and a condition.
 4. The method of claim 1,wherein the receiving the first identification information and thecapability information comprises receiving the first identificationinformation and the capability information based on user inputinformation, the user input information being received from theelectronic device and input by the user to the electronic device.
 5. Themethod of claim 1, wherein the received first identification informationand the received capability information are received separately from andindependently of the receiving the single user voice command.
 6. Themethod of claim 5, wherein: the received single user voice command doesnot include identifications of each of the identified at least twoexternal devices; and the identifying the at least two external devicescomprises analyzing the received single user voice command to determinethe at least two external devices corresponding to the received singleuser voice command.
 7. The method of claim 1, wherein the electronicdevice and the plurality of external devices are connected to a samenetwork via a home gateway.
 8. The method of claim 1, wherein thereceiving the single user voice command comprises receiving, by theserver from the electronic device via an Internet connection, the singleuser voice command input by the user to the electronic device.
 9. Themethod of claim 1, wherein the identifying the at least two externaldevices comprises, based on the server being unable to identify anexternal device based on the received single user voice command:requesting additional user input information from the electronic device;receiving, from the electronic device, an additional user voiceinformation input by the user to the electronic device; and identifyingthe at least two external devices based on the received additional uservoice information.
 10. The method of claim 1, wherein the generating thecontrol information for the control commands controlling the identifiedat least two external devices comprises generating the controlinformation for the control commands controlling the identified at leasttwo external devices in a predetermined order.
 11. A non-transitorycomputer-readable recording medium having recorded thereon a computerprogram which, when executed by a processor, causes the processor toexecute the method of claim
 1. 12. The method of claim 1, wherein theobtaining the user intent regarding the received single user voicecommand based on the received single user voice command comprisesperforming natural language processing on the received single user voicecommand.
 13. A first server comprising: a memory for storinginstructions; and a processor configured to execute the instructions to:obtain, from a second server, first identification information of aplurality of external devices and first capability information regardingcapabilities that the plurality of external devices are able to perform,obtain, from an electronic device distinct from the plurality ofexternal devices, a single user voice command input by a user to theelectronic device, obtain a user intent regarding the obtained singleuser voice command based on the obtained single user voice command,identify at least two external devices, from among the plurality ofexternal devices, based on the obtained user intent regarding the singleuser voice command and the first identification information of theplurality of external devices, generate control information for controlcommands controlling the identified at least two external devices, basedon the obtained user intent regarding the single user voice command,second capability information of the identified at least two externaldevices, and second identification information of the identified atleast two external devices, and output the generated control informationfor the control commands controlling the identified at least twoexternal devices.
 14. The first server of claim 13, wherein: the singleuser voice command is a phrase which is input into the electronic deviceby a user voice, and comprises a combinational instruction, and thecombinational instruction is directed to the electronic device andcomprises an action, a time, and a condition.
 15. The first server ofclaim 13, wherein the processor is configured to obtain the firstidentification information and the first capability information based onuser input information, the user input information being received fromthe electronic device and input by the user of the electronic device.16. The first server of claim 13, wherein the obtained firstidentification information and the obtained first capability informationare obtained separately from and independently of the obtaining thesingle user voice command.
 17. The first server of claim 16, wherein:the obtained single user voice command does not include identificationsof each of the identified at least two external devices; and theprocessor is configured to identify the at least two external devices byanalyzing the obtained single user voice command to determine the atleast two external devices corresponding to the obtained single uservoice command.
 18. The first server of claim 13, wherein the electronicdevice and the plurality of external devices are connected to a samenetwork via a home gateway.
 19. The first server of claim 13, wherein,based on the first server being unable to identify an external devicebased on the obtained single user voice command, the processor isconfigured to identify the at least two external devices by: requestingadditional user input information from the electronic device; obtaining,from the electronic device, an additional user voice information inputby the user to the electronic device; and identifying the at least twoexternal devices based on the obtained additional user voice informationinput.
 20. The first server of claim 13, wherein the processor isconfigured to generate the control information for control commandscontrolling the identified at least two external devices in apredetermined order.
 21. The first server of claim 13, wherein theprocessor is configured to obtain the user intent regarding the obtainedsingle user voice command based on the obtained single user voicecommand comprises performing natural language processing on the obtainedsingle user voice command.